Papers with logic reasoning
DivLogicEval: A Framework for Benchmarking Logical Reasoning Evaluation in Large Language Models (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing logic reasoning benchmarks are limited in language diversity and their distributions are deviated from ideal distributions, which may lead to biased evaluation results. |
| Approach: | They propose a new logic benchmark DivLogicEval that uses natural sentences to evaluate logical reasoning . |
| Outcome: | The proposed evaluation metric mitigates bias and randomness inherent in LLMs. |
ScratchEval: Are GPT-4o Smarter than My Child? Evaluating Large Multimodal Models with Visual Programming Challenges (2025.naacl-short)
Copied to clipboard
| Challenge: | Recent advances in large multimodal models (LMMs) have demonstrated impressive code generation capabilities, primarily evaluated through image-to-code benchmarks. |
| Approach: | They propose a visual programming reasoning benchmark based on Scratch, a block-based visual programming language widely used in children’s programming education. |
| Outcome: | The proposed framework evaluates the visual programming ability of large multimodal models by integrating visual elements and embedded programming logic. |
MWP-BERT: Numeracy-Augmented Pre-training for Math Word Problem Solving (2022.findings-naacl)
Copied to clipboard
| Challenge: | Existing work on math word problem solvers replace real numbers with symbolic placeholders to focus on logic reasoning. |
| Approach: | They propose to inject numerical properties into symbolic placeholders with contextualized representation learning schema to solve number representation dilemma. |
| Outcome: | The proposed model can solve MWP problems on English and Chinese benchmarks. |
Deep Inductive Logic Reasoning for Multi-Hop Reading Comprehension (2022.acl-long)
Copied to clipboard
| Challenge: | a new method for multi-hop reading comprehension uses deep learning to reason across documents . a previous study showed that deep learning methods only implicitly encode query-related information into embeddings . |
| Approach: | They propose a deep-learning based inductive logic reasoning method that extracts query-related information and conducts logic reasoning among filtered information. |
| Outcome: | The proposed model is evaluated on two reading comprehension datasets . it uses attentive memories with novel differentiable logic operators . |
PAGED: A Benchmark for Procedural Graphs Extraction from Documents (2024.acl-long)
Copied to clipboard
| Challenge: | Existing methods for extraction of procedural graphs from documents are not solving the task well. |
| Approach: | They propose a benchmark to test automatic extraction of procedural graphs from documents . they involve three advanced large language models and enhance them with a novel self-refine strategy . |
| Outcome: | The proposed benchmark systematically examines the progress of current methods and explores the potential of emerging large language models (LLMs) on this task. |
Investigating Transformer-Guided Chaining for Interpretable Natural Logic Reasoning (2023.findings-acl)
Copied to clipboard
| Challenge: | Natural logic reasoning has received increasing attention lately, with several datasets and neural models proposed, though with limited success. |
| Approach: | They propose to iteratively perform 1-step neural inferences and chain together the results to generate a multi-step reasoning trace. |
| Outcome: | The proposed method has high accuracies on a multi-hop First-Order Logic (FOL) reasoning benchmark. |
Disentangling Logic: The Role of Context in Large Language Model Reasoning Capabilities (2025.findings-acl)
Copied to clipboard
Wenyue Hua, Kaijie Zhu, Lingyao Li, Lizhou Fan, Mingyu Jin, Shuhang Lin, Haochen Xue, Zelong Li, Jindong Wang, Yongfeng Zhang
| Challenge: | Using large language models, large language model models can be used to evaluate reasoning abilities in context-rich scenarios. |
| Approach: | They construct datasets for both propositional logic and abductive logic reasoning with four difficulty levels across 12 distinct domains based on Wikipedia categorization and those with purely abstract variables. |
| Outcome: | The proposed model can be used to benchmark LLMs in real-world scenarios, but not in context-rich scenarios. |